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Abstract — In this paper, we introduce an efficient iterative 
solver for the joint linear-programming (LP) decoding of low- 
density parity-check (LDPC) codes and finite-state channels 
(FSCs). In particular, we extend the approach of iterative 
approximate LP decoding, proposed by Vontobel and Koetter 
and explored by Burshtein, to this problem. By taking advantage 
of the dual-domain structure of the joint decoding LP, we obtain 
a convergent iterative algorithm for joint LP decoding whose 
structure is similar to BCJR-based turbo equalization (TE). The 
result is a joint iterative decoder whose complexity is similar to 
TE but whose performance is similar to joint LP decoding. The 
main advantage of this decoder is that it appears to provide the 
predictability of joint LP decoding and superior performance 
with the computational complexity of TE. 

I. Introduction 

Iterative decoding of error-correcting codes, while intro- 
duced by Gallager in his 1960 Ph.D. thesis, was largely 
forgotten until the 1993 discovery of turbo codes by Berrou, 
et al. Since then, message-passing iterative decoding has been 
a very popular decoding algorithm in research and practice. In 
1995, the turbo decoding of a finite-state channel (FSC) and a 
convolutional code (instead of two convolutional codes) was 
introduced by Douillard, et al as a turbo equalization (TE) 
which enabled the joint-decoding of the channel and code 
by iterating between these two decoders [1|. Before this, one 
typically separated channel decoding from error-correcting 
code decoding [2][3 |. This breakthrough received immediate 
interest from the magnetic recording community, and TE 
was applied to magnetic recording channels by a variety 
of authors (e.g., ||4), 0, 0, Q). TE was later combined 
with turbo codes and also extended to low-density parity- 
check (LDPC) codes (and called joint iterative decoding) by 
constructing one large graph representing the constraints of 
both the channel and the code (e.g., [8]]). 

In ll9l lfT0l . Feldman, et al. introduced a linear- 
programming (LP) decoder for general binary linear 
codes and considered it specifically for both LDPC and 
turbo codes. It is based on solving an LP relaxation of an 
integer program which is equivalent to maximum-likelihood 
(ML) decoding. For long codes and/or low SNR, the 
performance of LP decoding appears to be slightly inferior 
to belief-propagation decoding. But, unlike the iterative 
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decoder, the LP decoder either detects a failure or outputs a 
codeword which is guaranteed to be the ML codeword. 

Recently, the LP formulation has been extended to the 
joint decoding of a binary-input FSC and outer LDPC code 
ifTTIl |[T2l . In this case, the performance of LP decoding ap- 
pears to outperform belief-propagation decoding at moderate 
SNR. Moreover, all integer solutions are indeed codewords 
and the joint decoder also has a certain ML certificate 
property. This allows all decoder failures to be explained by 
joint-decoding pseudo-codewords (see Fig. |2j. 

In the past, the primary value of LP decoding was as an 
analytical tool that allowed one to better understand iterative 
decoding and its modes of failure. This is because LP decod- 
ing based on standard LP solvers is quite impractical and has 
a superlinear complexity in the block length. This motivated 
several authors to propose low-complexity algorithms for LP 
decoding of LDPC codes in the last five years (e.g., fl3l . fl4l . 
lfI3I [16), Oil, ED, ED)- Many of these have their roots 
in the iterative Gauss-Seidel approach proposed by Vontobel 
and Koetter for approximate LP decoding [ 14 1. This approach 
was also studied further by Burshtein iTSl . 

In this paper, we extend this approach to the problem of 
low-complexity joint LP decoding of LDPC codes and FSCs. 
We argue that by taking advantage of the special structure in 
dual-domain of the joint LP problem and replacing minima in 
the formulation with soft-minima, we can obtain an efficient 
method that solves the joint LP. While there are many ways to 
iteratively solve the joint LP, our main goal was to derive one 
as the natural analogue of TE. This should lead to an efficient 
method for joint LP decoding whose performance is similar 
to joint LP and whose complexity similar to TE. Indeed, 
the solution we provide is a fast, iterative, and provably 
convergent form of TE and update rules are tightly connected 
to BCJR-based TE. This demonstrates that an iterative joint 
LP solver with a similar computational complexity as TE is 
feasible (see Remark [T0|l. 

The paper is structured as follows. After briefly reviewing 
joint LP decoding in Sec. [II] Sec.[III]is devoted to develop the 
iterative solver for the joint LP decoder, i.e., iterative joint 
LP decoder and its proof of convergence. Finally, we provide, 



in Sec. IV the decoder performance results and conclude in 
Sec.[V] Due to space limitations we omit many of the proofs, 
but they can be found in |20|. 



Figure 1, The dicode channel (DIC) is a binary-input FSC with a linear 
response of G(z) = 1 — z~ 1 and additive Gaussian noise. If the input bits 
are differentially encoded prior to transmission, then the resulting channel 
is called the preceded dicode channel (pDIC). The state diagrams of these 
two channels are shown: noiseless dicode channel without (left) and with 
precoding (right). The edges are labeled by the input/output pair. 

II. Background: Joint LP Decoder 

A. Notation 

Throughout the paper we borrow notation from 1101 , Let 
X = {1, . . . , N} and J = {1, . . . , M} be sets of indices 
for the variable and parity-check nodes of a binary linear 
code. A variable node i € X is connected to the set M(%) of 
neighboring parity-check nodes. Abusing notation, we also 
let M(j) be the neighboring variable nodes of a parity-check 
node j G J when it is clear from the context. For the trellis 
associated with a FSC, we let E — {1, . . . , O} index the set 
of trellis edges associated with one trellis section. For each 
edgd^ e G E N , in the length- N trellis, the functions t(e) — > 
{1, . . . , N}, s e -> S, s'(e) -> S, x(e) -> {0, 1}, and a e -> 
A map this edge to its respective time index, initial state, 
final state, input bit, and noiseless output symbol. Finally, 
the set of edges in the trellis section associated with time i 
is defined to be % = {e G E N | t(e) = i). 

B. Joint LP Decoder 

Now, we describe the joint LP decoder in terms of the 
trellis of the FSC and the checks in the binary linear cod^j 
Let N be the length of the code and y = (j/i, t/2> • • • > Vn) 
be the received sequence. The trellis consists of (N + 1)\S\ 
vertices (i.e., one for each state and time) and a set of at 
most 2A r |S| 2 edges (i.e., one edge for each input-labeled 
state transition and time). The LP formulation requires one 
indicator variable for each edge e G %, and we denote that 
variable by g i e . So, g i e is equal to 1 if the candidate path 
goes through the edge e in %. Likewise, the LP decoder 
requires one cost variable for each edge and we associate 
the branch metric b. L , e with the edge e given by 

A i-\nP(y tie) ,s , {e)\x(e),s(e)) ift(e)>l 
l - e \-ln[P(y t r e)> s'(e)\x(e),s(e))P (s(e))] ift(e) = l. 

Definition 1. The trellis polytope T enforces the flow con- 
servation constraints for channel decoder. The flow constraint 
for state k at time i is given by 



Fi, k = < g G [0, 1] 



NxO 



e:s' (e) — k 



e:s(e)— k 



'in this paper, e is used to denote a trellis edge while e denotes the 
universal constant that satisfies lne = 1. 

2 Extensions of this joint LP decoder to non-binary linear codes is 
straightforward based on [21]. 




Figure 2. Illustration of joint-LP decoder outputs for the single parity-check 
code SPC(3,2) over DIC (starts in zero state). By ordering the trellis edges 
appropriately, joint-LP decoder converges to either a trellis-wise (ML) code- 
word (0 1 0; I ; .0 1 0) (top dashed blue path) or a joint-decoding 
trellis-wise pseudo-codeword (0100;00.5.5;.5 0.5 0) (bottom dashed red 
paths). Using Q to project them into V(H), we obtain the corresponding 
(symbol-wise) codeword (1,1,0) and joint-decoding symbolwise pseudo- 
codeword (1, .5, 0). 

Using this, the trellis polytope T is given by 



N-l 



n n 



=1 fees 



= 1 , for any p 6 X 



Definition 2. Let Q be the projection of g onto the input 
vector f = (/i, . . . , f N ) G [0, 1} N defined by f = Qg with 



fi = 



eeTi-.x(e) = l 



Let C C {0, 1} N be the length- N binary linear code 
defined by a parity-check matrix and c = (ci, . . . , cjy) be 
a codeword. Let C be the set whose elements are the sets of 
indices involved in each parity check, or 

C = {Af(j)c{i,...,N}\jeJ}. 

Then, we can define the set of codewords to be 



C= ce{0,l} 



/V 



mod 2, VL G C 



The codeword polytope is the convex hull of C. This polytope 
can be quite complicated to describe though, so instead one 
constructs a simpler polytope using local constraints. Each 
parity-check L G C defines a local constraint equivalent to 
the extreme points of a polytope in [0, 1 



N 



Definition 3. The local codeword polytope LCP(L) asso- 
ciated with a parity check is the convex hull of the bit 
sequences that satisfy the check. It is given explicitly by 

LCP(L)^f) jcG[0, 1} N 5>-5><|S|-l). 

SQL { ieS i£L-S ) 

|S|odd 
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Problem-P: 

subject to 

^2 w i- B = 1 ' Vj e ff p> e = for anyp e - z: 

^2 Wj,B= ^2 9i, e , Viel,jeAf{i) 

Be£j,B3i e:x(e)=l 

- 9i - e = X! V« el\N, k E S 

e:s' (e) — fe e:s(e)— fe 

> 0, Vj ej,6e g 4 , e > 0, Viel,ee7^. 

Definition 4. The relaxed polytope V(H) is the intersection 
of the LCPs over all checks and 



P(Z0 = f) LCP(L). 



Lec 



Definition 5. The trellis-wise relaxed polytope Vf(H) for 
V{H) is given by 

V T (H)^{ger\QgeV(H)}. 

Theorem 6 ( ifTTI ). The LP joint decoder computes 

argmin V V, &;, e gi. e (1) 

and outputs a joint ML edge-path if g is integral. 

III. New Results: Iterative Joint LP Decoder 
A. Iterative Joint LP Decoder Derivation 

In this section, we develop an iterative solver for the joint 
decoding LP. There are few key steps in deriving our iterative 
solution for the joint LP decoding problem. For the first step, 
given by Problem-P, we reformulate the original LP ([1} in 
Thm. fusing only equality constraints involving the indicator 
variables^] g and w. 

The second step, given by Problem-Dl, follows from 
standard convex analysis (e.g., see f22\ ). The Lagrangian dual 
of Problem-P is equivalent to Problem-D 1 and the minimum 
of Problem-P is equal to the maximum of Problem-D 1 . From 
now on, we consider the Problem-Dl where the code and 
trellis constraints separate into two terms in the objective 
function. See Fig. [3] for a diagram of the variables involved. 

The third step, given by Problem-D2, observes that for- 
ward/backward recursions can be used to perform the op- 
timization over n and remove one of the dual variable 
vectors. This splitting was enabled by imposing the trellis 
flow normalization constraint in Problem-P only at one time 
instant p E 1. This detail gives N different ways to write 
the same LP and is an important part of obtaining update 
equations similar to TE. 

3 The valid patterns £j = {B C J\f (j) \ \B\ is even} for each parity-check 
j £ J allow us to define the indicator variables M>j,g (for j £ J and 
B 6 £j) which equal 1 if the codeword satisfies parity-check j using 
configuration B £ £j. 



Problem-Dl: 



max } min 

m,n Be£ 1 



.ieB 



-mm [r Pi6 — n p _ liS ( e )+n P)S '( e 



subject to 

r», e > 7li_l, a ( e ) - ™i, s '(e), Vi £ l\p, e € Ti 



and 
where 



no t k = n N , k = 0, Vfc e S, 

Ti.e — bi,e — <5x(e)=l ^ m 




^] W = {toi,b} e 



Figure 3. Illustration of primal variables g and w defined for Problem-P 
and dual variables n and m defined for Problem-Dl on the same example 
given by Fig. [5] SPC(3,2) with DIC for N = 3. 

Lemma 7. Problem-Dl is equivalent to the Problem-D2. 
Proof: By rewriting the inequality constraint in Problem- 

D as 

— n i,s'(ei) < ~ n i-l,s(ei) + ^i,e 

we obtain the recursive upper bound for i = p — 1 as 

< -n p - 2 , s (e p -i) + r V-l-eL'( e ^ 1 )=fc 

< _n p-3,s(e,,_3)+ r p-2,e| a ,( ) =g ( ) + rp_i )e | a/(e ^_ i j =J . 



p-1 



s'(e t ,_i)=/s^'(ep-2)=s(ep_i),.. ) s'(ei)=s(e2). 



This upper bound — n p -i t k < is achieved by the 

forward Viterbi update in Problem-D2 for i = 1, ... , p — 1. 
Again, by expressing the same constraint as 

we get recursive upper bound for i = p+1. Similar reasoning 
shows this upper bound n Pt k < "^~p,fc is achieved by the 
backward Viterbi update in Problem-D2 for i = N — 1, N — 
2, ... , p. See Fig. [4] for a graphical depiction of this. ■ 
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-mm[r Pi e- Vi )8 ( e )+n p , s '( e )] 
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Problem-D2: 

max y min 

m ^ Be£ 1 

where n i t k is defined for i = 1, ... , p — 1 by 

-~n i>k = min - ni_i, s ( e .) + r 4 . e , Vfc G 5 
and n,'^ is defined for i = N — 1, N — 2, ... . p by 

n, fc = min n i+1 s / (e ) + V i+ x e , VkeS 

ees- 1 (fc) 

starting from 



rt 



fe = n jv,fc = 0, Vfc € 5. 




-f*o,i = o 



tr 3il = o 



Figure 4. Illustration of Viterbi updates in Problem-D2 on the same example 
given by Fig. [2] DIC for N = 3 with forward it and backward . 

The fourth step, given by Problem-DS, is replacing mini- 
mum operator in Problem-D2 with the soft-minimum opera- 
tion. A smooth approximation is obtained by using 



min(xi, x 2 , 



1 



K 



i=l 



as in fm . It is easy to verify that this log-sum-exp function 
converges to the minimum function as K increases. Since 
the soft-minimum function is used in two different ways, we 
use different constants, K x and K 2 , for the code and trellis 
terms. This Problem-DS allows one to to take derivative 
of (|2]l (giving the KKT equations, derived in Lemma [8), 
and represent |3]l and |4} as BCJR-like forward/backward 
recursions (given by Lemma [9]). 

Lemma 8. The unique maximum of |2} over {mpj'jj'gjv'fp) 
can be found using the KKT equations, and an iterative 
solution for p E I is given by 



i p jt — M p j> 



M V1 , 4 — In 1 lp ' j ' 



for f G Af(p) where 



pj> = J~J tanh 

ieJV"(j')\p 



KlTTli 



7 P 



4 In 



E 



e£7~ p :x(e)=0 



i, s ( e ) 



w) 



E 



eS7^,:a;(e) = l 



,-Ar2(r p -Tt p _ lj3(e) +fr p a , (e) ) ' 



Lemma 9. Equations ([3]) and Q are equivalent to the BCJR- 
based forward and backward recursion given by Q, |6]), and 
0. 



Problem-DS: 

max ~^H ln Z! e- Xl {^^^o) m - lB(l) } (2) 



is/ Befj 



— In V e - Kj { r ^«-^-MW+ fr P,»'w} 



where lg (i) is the indicator function of the set £>, n^k is 
defined for i = 1, ... , p — 1 by 

-^ = -^ln 2 e-M-^J+M, (3) 

and n^k is defined for i = N — 1, N — 2, ... , p by 

W fe = -^m E e -^{ fe — <- +1 > +r — } (4) 



e (+i es- 1 (fc) 



starting from 



^o,fc = ^w,fc = 0, Vfc e S. 



Now, we have all the pieces to complete the algorithm. As 
the last step, we combine the results of Lemma [8] and [9] to 
obtain the iterative solver for the joint decoding LP, which 
is summarized in Algorithm [1] 

Remark 10. While resulting Algorithm [1] has the bit-node 
update different from standard belief propagation (BP), we 
note that setting K\ = 1 in the inner loop gives the exact 
BP check-node update and setting K2 = 1 in the outer loop 
gives the exact BCJR channel update. In fact, one surprising 
result of this work is that such a small change to the BCJR- 
based TE update provides an iterative solver for the LP whose 
complexity similar to TE. It is also possible to prove the 
convergence of a slightly modified iterative solver that is 
based on a less efficient update schedule. 

B. Convergence Analysis 

This section considers the convergence properties of the 
proposed Algorithm [1] Although we have always observed 
convergence of Algorithm [T] in simulation, our proof requires 
a modified update schedule that is less computationally 
efficient. Following Vontobel's approach in lfl4l . which is 
based on general properties of Gauss-Seidel-type algorithms 
for convex minimization, we show that the modified version 
Algorithm [T] is guaranteed to converge. Moreover, a feasible 
primal solution can be obtained that is arbitrarily close to the 
solution of Problem-P. 

The modified update rule for Algorithm [T] consists of 
cyclically, for each p = 1, . . . , N, computing the quantity 
7 P (via step 2 of Algorithm [Tji and then updating m Pt j for 
all j g jV(p) (based on step 3 of Algorithm [TJ. The drawback 
of this approach is that one BCJR update is required for each 
bit update, rather than for N bit updates. This modification 
allows us to interpret Algorithm [1] as a Gauss-Seidel-type 
algorithm. Therefore, the next theorem can be seen as a 
natural generalization of |14||18|. 

Theorem 11. Let P* and P be the minimum value of 



Algorithm 1 Iterative Joint Linear-Programming Decoding 

• Step 1. Initialize rriij = for i £ I, j £ and 
iteration count I = 0. 

• Step 2. Update Outer Loop: For i £ I, 

- (i) Compute bit-to-trellis message 



where 



E 



- (ii) Compute forward/backward trellis messages 

a t+ i(fc)= rr^r^ (5) 

E fc E ees '-i(fc) a * ( s ( e ^ ' Al +^ e 
fl ^ £ee s - 1(fc )&(s'(e))-A t , e 
EfeEee s -i(fe) «( s (e)) • A V 

where ^ (k) =a (k) = l/ \S\ for all k £ S. 

- (iii) Compute trellis-to-bit message ji 

. Eee"n:x( e )=0 a i~l ( S ( e )) A i,eft (s'(e)) 
EeeT 4 :x(e)=l a i-l ( S ( 6 )) KePi (s'(e)) 

Step 3. Update Inner Loop for ^ mnel - rounds: For i £ 1, 

- (i) Compute bit-to-check msg rriij for j <E AT (i) 

rriij = Mij + -j^- 
Ki 

- (ii) Compute check-to-bit msg Mjj for j £ M (i) 
1 , l-h 



where 



Mi i = — In * ^ 

J 1 + li,,- 



k,j = n tanh ( Xi ^ r,j 



r£Af(j)\i 

Step 4. Compute hard decisions and stopping rule 
- (i) For i e 2, 



fi 



1 if 7 i < 
0, otherwise 



- (ii) If f satisfies all parity checks or the iteration 
number, ^outer> i s reached, stop and output f. 
Otherwise increase £ by 1 and go to Step 2. 

Problem-P and Problem-PS^] and denote g be the optimum 
solution of Problem-PS. For any S > 0, there exist suffi- 
ciently large K\ and Ki such that sufficient many iterations 
of the modified Algorithm [T] yields g which is feasible in 
Problem-P and satisfies 

P - P* 
— ^' 

4 To show connections between problem descriptions clearly, we write 
the Lagrangian dual of Problem-DS as Problem-PS by letting Wj = 

{ w j,b} b&£ ., 9v — {9p,e} e( zT p and H ( x ~> - ~J2i x i lnx i for x m the 
standard simplex. The minimum of Problem-PS is equal to the maximum 
of Problem-DS. 



Problem-PS: 



W j 



iel eGTi j£j 

subject to the same constraints as Problem-P. 



H(9p) 



IV. Performance of Iterative Joint LP Decoder 

To validate proposed solutions for the problem of the joint 
decoding of a binary-input FSC and outer LDPC, we use the 
following two simulation setups: 

• For preliminary studies, we use (3, 5)-regular binary 
LDPC codes with length 455 on the precoded dicode 
channel (pDIC) 

• For practical study, we use a (3, 27)-regular binary 
LDPC code with length 4923 and rate 8/9 on the class- 
II Partial Response (PR2) channel used as a partial- 
response target for perpendicular magnetic recording. 

All parity-check matrices were chosen randomly except that 
double-edges and four-cycles were avoided. Since the per- 
formance depends on the transmitted codeword, the results 
were obtained for a few chosen codewords of fixed weight. 
The weight was chosen to be roughly half the block length, 
giving weights 226 and 2462. 

Fig. [5] shows the decoding results based on the Algorithm 
[T] compared with the joint LP decoding performed in the 
dual domain using MATLAB in the first setup. The choice 
of parameters and scheduling scheme has yet to be optimized. 
Instead, we use a simple scheduling update scheme: variables 
are updated cyclically with 5 inner loop iterations after single 
outer iteration with K\ = 1000 and K% = 100. Somewhat 
interestingly, we find that iterative joint LP decoding WER 
curve loses about 0.2 dB at low SNR. This may be caused 
by using too few iterations or finite values of K x and K 2 - 
But, at high SNR this gap disappears and the curve converges 
towards the error rate predicted for joint LP decoding. This 
shows that joint LP decoding outperforms belief-propagation 
decoding for short length code at moderate SNR with the 
predictability of LP decoding. Of course this can be achieved 
with a computational complexity similar to turbo equaliza- 
tion. 

Fig. [6] shows the decoding results based on the Algorithm 
[T] compared with the state -based JIMPD algorithm described 
in 1 23 1 in more practical scenario. To make a fair comparison, 
we fix the maximum iteration count, ^ uter (^inner + ■"•) 
of each algorithm to roughly 1000 and choose K\ — 
1000 and K 2 = 10 for Algorithm [T] Surprisingly, we find 
that iterative joint LP decoding WER curve with Algorithm [T] 
wins over JIMPD at all SNR with substantial gains. Also, the 
slope difference between two curves anticipate greatly better 
error-floor performance of Algorithm [T] This shows that 
joint LP decoding outperforms belief-propagation decoding 
even for long length code at all SNR with a computational 
complexity similar to TE. 

V. Conclusions 

In this paper, we consider the problem of low-complexity 
joint linear-programming (LP) decoding of low-density 



Regular (3,5) code of length 450 




•-IJLPD WER 
- JIMPD WER 
■-JLPD WER1 
• JLPDWER2 
♦-JLPD WER3 
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SNR(dB) 
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Figure 5. This figure shows comparison between the joint LP decoding 
(JLPD), joint iterative message-passing decoding (JIMPD), and iterative joint 
LP decoding (IJLPD) on the pDIC with AWGN for random (3,5) regular 
LDPC codes of length N = 450. The curves shown are the JLPD WER 
(solid), JLPD WER prediction (dashed), JIMPD WER (dash-dot), and IJLPD 
WER (circle-solid). The dashed curves are computed using the union bound 
based on joint-decoding pseudo-codewords observed at 2.67 dB as described 
in 1 11 1 and the dash-dot curves are obtained using the state-based JIMPD 
described in 1231 . The circle-solid curves are computed using Algorithm^ 
Note that SNR is defined as channel output power divided by a 2 . 

parity-check codes and finite-state channels. We present a 
novel iterative solver for the joint LP decoding problem. This 
greatly reduces the computational complexity of the joint 
LP solver by exploiting the LP dual problem structure. Its 
main advantage is that it provides the predictability of LP 
decoding and significant gains over turbo equalization (TE) 
with a computational complexity similar to TE. 
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